Flash storing

Introduction

With this implementation, the Smartcitizen kit 2.1 readings will be stored in a flash memory buffer before being published. If SCK RTC clock is in sync the kit will start taking readings and save them, if a SD card is available it will also save them there and if the Wi-Fi and platform token are configured it will publish the data to the network via MQTT. This means we will always have a backup for the readings taken the last ~70 days.

Info

The memory chip that we use (S25FL064L), it's an 8 MB SPI flash nonvolatile memory. To manage it, we use the SPI Memory Arduino library.

Data organization

Flash sectors

The minimum erasable unit is a sector (4 KB), so we use this as a base for data organization. On the first 3 bytes of each sector we store information about its contents, in this way we can find which ones are empty, currently in use or have valid data.

enum SectorAddr {
    SECTOR_STATE = 0x00,
    SECTOR_NET   = 0x01,
    SECTOR_SD    = 0x02
};

The first byte of each sector (SECTOR_STATE) describes if the sector is empty or already used. On boot, we scan sector by sector until we find an empty one to store new data.

The SECTOR_NET and SECTOR_SD bytes of each sector contain the flags related to the publish state of contained data. This bytes are only marked if all the groups of the sector are already published and the sector is full.

When all flash sectors are marked as used we will erase the oldest data sector and reuse it.

Data Groups

Readings are grouped by the time there were taken and saved sequentially inside flash memory sectors. On the beginning of each group, we store information about it.

enum GroupAddr {
    GROUP_SIZE =        0x00,
    GROUP_NET  =        0x02,
    GROUP_SD   =        0x03,
    GROUP_TIME =        0x04,
    GROUP_READINGS =    0x08
};

In the GROUP_SIZE bytes, we store the size in bytes of the full group with all its contents. This allows us to start reading the sector starting on the first group and jump from group to group very fast.

The GROUP_NET and GROUP_SD bytes of each group contain the flags related to the publishing state of contained data, as default they are in NOT_PUBLISHED state until we set them to PUBLISHED.

The time and date of when these readings were taken is stored in the GROUP_TIME bytes in Epoch time format.

Starting on the GROUP_READINGS byte, we store the sensor readings data (256 max). The first byte of each reading contains its full size in bytes, the second byte the sensor identifier (SensorType) and starting on the third byte the sensor data is stored in ASCII chars.

Sizes

A normal reading group with the default urban board hardware installed is composed by 11 readings, we expect each reading to take 7 bytes: an average of 5 bytes for the reading itself plus 2 overhead bytes for SensorType and size, so each group should have a total of 77 bytes of readings, 2 bytes of size, 2 bytes of flags and 4 bytes of the time stamp. That means we can expect a normal group to be around 85 bytes.

Info

We have 8 MB (8,388,608 bytes) of flash memory that means storing almost 100,000 groups of readings or around 70 days of readings with standard sensor hardware. This number can vary a little, because we will lose some space at the end of each sector.

About flash memory lifespan, rounding numbers we can say we have enough space to store 2 months (60 days) of readings, according to the Flash memory datasheet we have at least 100,000 erase cycles: 2 months per cycle means 200,000 months that's more than 16k years!!.

User interface

Some aspects of the flash memory can be managed by the user via the SCK shell interface, issuing the help command you can see a brief description of the flash command interface:

flash:       Shows and manage flash memory state [no-param -> info] [-format (be carefull)] [-dump sect-num (0-2040)] [-sector sect-num] [-recover sect-num/all net/sd]

Without any parameter, the flash command will scan the memory and print out a table of its contents, showing totals at the end. Be patient, scanning can take a long time if your flash has data.
The -format option will erase the full flash memory.
The -dump option will dump (in hexadecimal values) the content of the requested sector.
The -sector option will show general information about the data contained on the requested sector.
The -recover option takes 2 parameters: the sector number or the keyword all and one keyword to indicate how to recover the data (net or sd)